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ABSTRACT 


Access control to sensitive information is a vital concern for 
Department of Defense agencies. Current methods employed to 
control access are vulnerable to unauthorized users and frequently 
inadequate. The use of biometric access control devices, such as 
Signature verification systems, may represent a solution to the 
access control problem. This thesis looked at two dynamic 
signature verification systems and compared their performance in 
general as well as under the different operating conditions of 
lined and unlined paper and morning and afternoon use. The two 
signature verification systems were the CIC system and the Sign/On 
system. Additionally, the thesis compared the CIC system under 
both sets of operating conditions using an inking stylus pen and a 
non-inking stylus pen. The experimental results indicated there 
was no significant difference between the CIC system using an 
inking stylus pen and the Sign/On system and that both systems had 
Type I error rates of less than 3% and Type II error rates of less 
than 1%. The results also indicated that the operating conditions 


test did not favor either system. 
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I. INTRODUCTION 

Access control to sensitive information is a vital concern 
for Department of Defense agencies. Because the cost of 
unauthorized access is often intolerable, the techniques used 
to control both physical access and computer access must be 
closely examined. In the interest of national security as 
well as personal privacy protection, adequate safeguards 
against unauthorized access must be guaranteed. 

Access control is the security function whereby a user is 
identified and screened to discriminate authorized from 
unauthorized use (MIL-HNBK-1031/1, 1987, p. 114). There are 
two different types of personal identification and screening 
involved in access control (FIPS Pub 83, 1980, p. 9). The 
first of these is the actual establishment of identity. [In 
this case it is the controlling system which is responsible 
for the identification of the person wishing access. The 
other alternative is to simply verify a claimed identity. A 
potential user would enter a claimed identity and then perform 
other system related tasks to verify the claim. Access would 
be granted or denied based upon the verification of identity 
rather than on the establishment of identity. Typically it is 
much faster and easier to verify an identity rather than 


establish an identity. 


There are three basic methods used to verify a person's 
identity for access control (FIPS Pub 48, 1977, p. 7). These 
methods are: 

1. Something a person knows. 

2. Something a person possesses. 

3. Something about a person. 

The first method relies on such things as passwords, lock 
combinations, or other pieces of personal information to 
verify identity. The most common is the pasSword. There are 
problems with this method in that the authorized user may 
forget the known information and be denied access or an 
unauthorized user may obtain the known information and be 
granted access. In either situation, access control was not 
effective. 

The second verification method relies on such things as 
keys, badges, or passes. The degree of security provided by 
this method is limited, however, because possessions can be 
lost, stolen or duplicated. Additionally, accounting for each 
key or badge can be an administrative burden in the ever 
changing environment usually found in the Department of 
Defense. 

Because of the inherent vulnerabilities in the other two 
methods, physiological attributes are being considered as 
possible bases for personal identification. Using 
physiological attributes for personal identification is one 


area in the field of study called biometrics. Some of the 


physical attributes currently considered for access control 
are voice, retinal blood vessel patterns, signature dynamics, 
typing dynamics, three dimensional hand geometry, and 
fingerprints. The critical problem with biometric devices 
used for identity verification is the difficulty in performing 
precise, repeatable measurements on the human body (FIPS Pub 
48, 1977, p. 10). This is especially true for the class of 
biometric devices which require a dynamic measure like speech 
or handwriting rather than a relatively static measure like a 
fingerprint. Another concern with biometric devices is the 
amount of interpersonal variation versus intrapersonal 
variation (FIPS Pub 83, 1980, p. 9). Interpersonal variations 
are those exhibited among different individuals whereas 
intrapersonal variations are those exhibited from the same 
individual from one measurement to the next. Introducing 
intrapersonal variations requires a much more sophisticated 
biometric device in order to verify identity. 

Signature verification machines are one type of biometric 
device being considered for access control. Handwriting 
generally presents unique characteristics from one individual 
to the next. The signature, however, is even more unique and 
personally stylized because of a lifetime of practice. Often 
the signature is similar to a conditioned reflex and is 
written with little attention paid to each individual letter 


Ge Leo werubpe se aelos7 7, pe Ll). The end result of this 


combination of features produces a highly individualized 
biometric measure which is very difficult to duplicate. 

There are two basic methods used in signature verification 
systems (FIPS Pub 83, 1980, p. 33). One of these involves the 
comparison of a static signature. An example is a scanning 
device which measures only the written form of the signature. 
This method is extremely vulnerable to deceit. Another method 
compares a dynamic signature with a reference signature. In 
other words, the potential user submits a "live" sample 
signature which is compared to a reference Signature. What 
this system measures has nothing to do with the signature's 
appearance, but with the forces generated by the user with the 
stylus or writing instrument (Bakke, 1986, p. 113) This 
method takes full advantage of how the signature is created. 
It is much more difficult to duplicate the dynamic motions 
used to create a signature than it is to duplicate the 
finished product. 

There are many ways to obtain the signals which represent 
a dynamic signature. The quantities measurable during the 
writing process are positions, forces, and accelerations. 
Instrumentation designed to measure these quantities may 
involve the writing instrument (stylus) or writing surface 
(platen) or a combination of the two (FIPS Pub 83, 1980, p. 
33). The vector of these "live" measurements is compared to 
the reference signature vector to verify identity. In theory, 


other written words may be used instead of the signature to 


verify identity, however, signatures are preferable because of 
the level of conditioned response involved. 

There are two dynamic signature verification systems in 
the Man-Machine Systems Design Laboratory at the Naval 
Postgraduate School in Monterey, California. One of them is 
produced by the Communication Intelligence Corporation and is 
called On-line Dynamic Signature Verification. This system 
consists of the stylus, the platen, and the software 
algorithms. Currently the system is connected to a personal 
computer for operation. The other system is produced by 
Capital Security Systems Incorporated and is called Sign/On. 
This system consists of the stylus, the platen, and a 
controlling unit with the software algorithm. The Sign/On is 
connected to a monitor for use. The purpose of this thesis is 
to design an experiment to evaluate and compare these two 
systems, analyze the statistical results, and report the 


findings. 


Il. THE EXPERIMENT 


This experiment compared the effectiveness of two 
signature verification systems under different operating 
conditions. One of the systems was produced by the 
Communication Intelligence Corporation (CIC) and was called 
On-line Dynamic Signature Verification. The other signature 
verification system was produced by Capital Security Systems 
Incorporated and was called Sign/On. 

The different operating conditions being compared were 
lined paper versus unlined paper, morning versus afternoon 
use, and an inking stylus versus a non-inking stylus pen. 
Only the CIC system was capable of operating with a non-inking 
stylus and therefore, the inking stylus pen versus the non- 
inking stylus pen condition was compared for that system 
alone. The comparison of these two systems under the 
different operating conditions may suggest preferred operating 
conditions for better performance and may indicate a more 


effective system. 


A. THE EQUIPMENT 
Both the Communication Intelligence Corporation and 
Capital Security Systems Incorporated were contacted to 


request specific information concerning the measurements 


recorded for verification. Neither company chose to respond 
to the requests. Operating procedures for both systems are 
described below. However, generic information provided by the 
companies indicated each system works on the speed of 
acceleration and deceleration of the stylus movements and not 
on the pressure applied to the stylus. 
1. The CIC System 

When first approaching the CIC system the user is 
prompted by the monitor screen to select a mode of operation 
from the main menu. The choices are either enroll or verify. 
The user touches the stylus to the enroll or verify box 
(Figure 1) depending on the user's wishes. The monitor screen 
then prompts the user to enter his personal identification 
number (PIN) in the scratchpad. The user must print each of 


the alphanumeric characters in a separate box in the 


scratchpad. The monitor screen echoes the input from the 
scratchpad. The system must correctly identify the 
characters in order to retrieve the correct file. Ef ca 


character recognized by the system is incorrect, the user 
Simply writes the character over in the same box until it is 
recognized correctly. Once the PIN is correctly recognized, 
the monitor screen prompts the user to touch the continue box 
on the writing surface with the stylus pen for faster 
response. The system now retrieves the proper file and 


Signature verification can begin. 


The user is now prompted by the monitor screen to 
submit a signature. As before the user is asked to touch the 


continue box for faster response. After completion of the 


Figure 1. CIC SYSTEM WRITING SURFACE 
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signature, the system compares the submitted signature to the 
reference signature for verification. Depending upon the 
comparison, the system may either use the monitor screen to 
prompt the user for up to two more Signatures or make a 
verification decision. The system will prompt the user for 
another signature if the comparison of the signatures is 
"close" but not exact. However, after a total of three 


submitted signatures a verification decision must be made. If 


the comparison is vastly different or exact, the verification 
decision can be made immediately. The results of the decision 
will be displayed on the monitor screen. The monitor screen 
will show the user that the system accepted the claimed 
identity or that the system recognized a forgery. At this 
time the user touches the quit box with the stylus to return 
to the main menu. 
2. The Sign/On System 

When first approaching the Sign/On system, the monitor 
screen and an illuminated light above ENTER ID on the writing 
surface prompt the user for a PIN (Figure 2). The user types 
the PIN on the monitor's keyboard. The writing surface has a 
series of red lights to prompt the user from this point until 
completion of the access attempt. After the PIN has been 
entered, the light above SIGN ON on the writing surface is 
illuminated. The user then submits a Signature on the tablet. 
Based on a Single signature the verification decision is made. 
If the system verified the claimed identity, the DOOR OPEN 
light illuminates. If the system rejects the claimed 
identity, all the lights above the writing surface flash. 
Additionally, the monitor screen duplicates the lighted 
verification decision. The system is then ready to prompt the 


next user for a PIN. 


Figure 2. SIGN/ON SYSTEM WRITING SURFACE 
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B. PERFORMANCE MEASURES 

The purpose of the experiment was to compare the 
performance of these two signature verification systems under 
different operating conditions. In order to accurately 
measure performance, two types of errors could be identified 
(Holmes, 1990, p. 2). They were: 

1. Type I Errors: Falsely rejecting an authorized user. 

2-e Y PCa -ERrOns: Falsely accepting an unauthorized 

user. 

The perfect biometric device can positively distinguish 

interpersonal variances from intrapersonal variances and thus 


eliminate both Type I and Type II errors. In other words if 
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the biometric device can accurately identify intrapersonal 
variances, the result is a decline in the number of Type I 
errors. Similarly, if the device can correctly recognize 
interpersonal variations, Type II errors decrease. However, 
if the system is not perfect, it must adequately deal with 


both types of errors. 


C. SUBJECTS 

In all, 24 subjects were used in the experiment. They 
were all military officers at the Naval Postgraduate School 
(NPS) in Monterey, California. All were pursuing graduate 
degrees at the time of the testing. There were 5 females and 
19 males ranging in age from 27 to 44 participating. None 
were familiar with signature verification equipment prior to 
testing and no incentives were given for participation. The 
two systems were set up in the Man-Machine Systems Design 
Laboratory at NPS and were available to the subjects from 0800 
to 1700 each weekday. Testing was unsupervised to create a 
more realistic environment and data sheets recording the 
success or failure of each access attempt were collected each 
week of testing. The subject sample was considered 
representative of the population of military members requiring 


access to controlled areas and information. 
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D. ENROLLMENT 

The first part of the experiment was the enrollment 
process. This process was the procedure by which the 
Signature verification system converts a subject's signature 
into a reference template. A reference template is a sample 
of the subject's signature stored in memory to which an actual 
Signature is compared when access is attempted. If, based on 
this comparison, the system verifies the subject's identity, 
access will be granted. Otherwise, the subject will be denied 
access. 

Each subject was guided through the enrollment procedure 
by the experimenter. For the CIC system each subject was 
assigned a three letter PIN. The first action required by 
each subject was to select the enroll mode. This was done by 
touching the stylus pen to the enroll box on the system 
writing surface (Figure 1). Next the subject was prompted to 
enter his PIN in the system scratchpad. The subject was then 
prompted to write six signatures. After each signature the 
subject was given the choice to submit or reject the 
Signature. Six submitted signatures were required for 
enrollment. The monitor then prompts the subject for one more 
Signature to verify enrollment. At this time the signature 
template was created and stored in memory. To test the 
accuracy of the enrollment procedure and to familiarize the 
subject with the system, each subject was required to gain 


access three consecutive times. If any of these three 
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attempts were unsuccessful, the subject repeated the 
enrollment procedure from start to finish until three 
successful attempts to gain access were recorded. 

The enrollment procedure for the Sign/On system was also 
guided by the experimenter. Each subject must again create a 
reference template. To create the template on the Sign/On 
system each subject first entered an assigned three digit PIN 
on the monitor keypad. The system illuminates the light above 
SIGN ON on the writing surface (Figure 2) to prompt the 
subject to submit two signatures. Thus the template was 
created and stored in the system's memory. Again each subject 
was required to gain access three consecutive times to become 
familiar with the system and to test the accuracy of the 
enrollment procedure. 

Both systems used lined paper and inking stylus pens for 
enrollment. Half the subjects were enrolled in a morning 
session and the other half in an afternoon session. An 
enrollment procedure similar to the one described above might 


be appropriate for new users in an operational environment. 


E. TESTING PROCEDURES 

The experiment ran for a period of 10 weeks following the 
enrollment process. It was conducted in 4 phases. Phase 1 
consisted of access attempts on both systems testing the 
effects of lined and unlined paper and morning and afternoon 


use. The CIC system allows the subject to write his signature 
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up to three times before access iS granted or denied. 
However, if the signature is vastly different from the 
template on either of the first two signatures, access is 
immediately denied. In other words because the first 
Signature is vastly different from the template stored in the 
CIC system's memory, three signatures are not permitted and 
access is immediately denied. With the Sign/On system the user 
is allowed only one Signature after which a verification 
decision 1s made. In order to compare the systems for this 
experiment, however, the Sign/On system was adjusted so that 
the subject could attempt access with up to three signatures. 
A success on either system was defined to be access granted on 
any of three signatures. 

The subjects were asked to attempt access four times per 
Gay on each system both in the morning and in the afternoon 
for three weeks. In total 16 signatures were collected from 
each subject each day. The systems were alternated between 
lined and unlined paper ona daily basis. The morning test 
period ran from 0800 to 1200 and the afternoon period ran from 
1201 to 1700. The systems were available for four days per 
week to facilitate both paper conditions (lined and unlined 
paper). Subjects were asked to alternate their first access 
attempt each day between the two systems to eliminate any 
possible effects due to ordering. 

Phase 2 of the experiment concerned falsely accepting 


unauthorized users, or Type II errors. This is also called 
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impostor testing. A list of subject identification codes for 
both systems as well as a sample signature was posted. Each 
subject was asked to attempt access uSing every other 
subject's signature. The impostor, or subject using another's 
identification code, attempted access on lined paper in the 
morning and afternoon and on unlined paper in the morning and 
afternoon. Each impostor attempted access in every other 
account one time for both time conditions and both paper 
conditions. Access was granted or denied on both systems 
based on up to three signatures per impostor depending on the 
comparison between the submitted Signature and the reference 
template. Because of the difficulty in imitating another's 
Signature, it was assumed the differences between the two 
systems in the number of signatures allowed before access was 
granted or denied would not matter. More specifically, the 
impostor Signature and the reference template were assumed to 
be so different that the CIC system would make a verification 
decision on the first attempt. In total 24 impostor 
Signatures per subject for each condition were collected. 
Phase 3 was a replication of Phase 1 but only using the 
CIC system with the non-inking stylus pen. All subjects were 
re-enrolled in the CIC system as before. Lined paper was used 
for the re-enrollment with half the subjects re-enrolled ina 
morning session and the other half in an afternoon session. 
The experiment proceeded as described in Phase 1 with the CIC 


system and the non-inking stylus. 
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Phase 4 of the experiment was identical to Phase 2 using 
the non-inking stylus pen on the CIC system alone. 

The data collected from all four phases of the experiment 
was analyzed to compare performance of the two signature 
verification systems. The Sign Test was used to compare the 
systems and a confidence interval was used to estimate the 


proportion of successful access attempts for both systems. 
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III. RESULTS AND DISCUSSION 


After analyzing the data collected from the experiment, 
the Sign Test was determined appropriate to compare (1) the 
CIC and the Sign/On signature verification systems, (2) the 
CIC signature verification system with an inking and a non- 
inking stylus, (3) the CIC and the Sign/On systems for 
impostor testing, and (4) the CIC system with an inking stylus 
and a non-inking stylus for impostor testing. Two separate 
series of tests were completed for each of the four situations 
above. The first series consisted of 4 sets of four tests 
each and compared each system under the different operating 
conditions (lined and unlined paper; morning and afternoon 
use). The second series consisted of 4 tests that combined 
all four operating conditions and re-compared the systems. 
In all twenty different Sign Tests were completed. 
Additionally, three confidence intervals were calculated to 
estimate the proportion of successful access attempts for the 
CIC system with an inking stylus, the Sign/On system, and the 
CIC system with a non-inking stylus. Three more confidence 
intervals were calculated to estimate the proportion of 
successful access attempts during impostor testing for the CIC 
system with an inking stylus, the Sign/On system, and the CIC 


system with a non-inking stylus. 
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A. DATA COLLECTED 

Due to unforeseen scheduling conflicts, seven subjects 
were unable to attempt access the requested total of 24 times 
for each operating condition. These seven subjects did, 
however, attempt access a total of 20 times for each operating 
condition. Because the data used to compute the statistics 
were proportions, the difference due to sample size between a 
proportion of 24 samples and one of 20 samples was considered 
negligible. Additionally, only 18 subjects participated in 
the phases of the experiment using the CIC system with the 
non-inking stylus. The smaller sample size was used in the 
comparison of the CIC system with an inking and a non-inking 


stylus. 


B. SIGN TEST 

The Sign Test is used in testing to determine if one 
random variable ina pair (X,Y) tends to be larger than the 
other random variable in the pair (Conover, 1980, p. 122). 
For this experiment, the random variable pairs were 
proportions of successful access attempts. The basis for 


pairing the observations was the common subject for each 


proportion. The proportions within the pair were not 
independent, however, each bivariate random pair was 
independent of the others. In addition, the pairs were 


consistent for each separate test. 
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1. The CIC System vs. The Sign/On System 

For the four Sign Tests comparing the different 
operating conditions for the CIC system and the Sign/On 
system, the X's were defined to be the proportion of 
successful access attempts on the CIC system and the Y's were 
defined to be the proportion of successful access attempts on 
the Sign/On system. The test and the results from the first 
Sign Test comparing the two systems on lined paper in the 
morning are summarized in Table 1. The test clearly 
demonstrated that the combined operating conditions of lined 
paper and morning use did not significantly favor one system 
over the other. The significance level was 0.01. 

The remainder of the Sign Tests and their results are 
in Tables 2 - 15 in the Appendix. Table 2 summarized the test 
and the results from the second Sign Test for the comparison 
of the CIC system and the Sign/On system under lined paper in 
the afternoon. Again the combination of operating conditions 
was consistent among the systems and did not indicate any 
difference between the two systems with a significance level 
or (0:01. 

The results from third Sign Test are similarly 
summarized in Table 3. No significant differences between the 
performances of the CIC system and the Sign/On system were 
indicated by combining the operating conditions of unlined 


paper and morning use. The significance level was 0.01. 
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Table 1. SIGN TEST FOR THE CIC SYSTEM vs. THE SIGN/ON 
SYSTEM ON LINED PAPER IN THE MORNING 


Cre SIGN/ON DIFFERENCE 
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The hypotheses are: 


Ho: The CIC system and the Sign/On system have the same 
proportion of successful access attempts on lined paper 
in the morning 


Ha: The CIC system and the Sign/On system do not have 
the same proportion of successful access attempts on 
lined paper in the morning 


0.01 


The observed number of minus signs is 3 out of a 
possible 9. From any Binomial Distribution Table, the 
corresponding probability level is 0.2539. 


Because the probability level is greater than «a, one 
CANNOT REJECT THAT THE PROPORTION OF SUCCESSFUL ACCESS 
ATTEMPTS FOR BOTH SYSTEMS ARE THE SAME ON LINED PAPER IN 
THE MORNING AT A SIGNIFICANCE LEVEL OF 0.01. 


* only subjects with differences not equal to O were 
considered 





The final Sign Test comparing the CIC system and the 
Sign/On system under the different operating conditions 
compared the systems under unlined paper with afternoon use. 
The test and results are summarized in Table 4. There was no 
significant difference between the two systems at a 


Significance level of 0.01 on unlined paper in the afternoon. 
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In summary, neither of the operating conditions of 
paper nor time significantly favored either the CIC system or 
the Sign/On system. It should be noted, however, that because 
the same subjects were used in all the Sign Tests, the tests 
and their respective results are not independent. 

2. The CIC System with Inking Stylus vs. The CIC System 
with Non-Inking Stylus 

A second set of four Sign Tests was performed to 
compare the different operating conditions for the CIC system 
with an inking stylus and the CIC system with a non-inking 
stylus. In these four test the X's were defined to be the 
proportion of successful access attempts using an inking 
stylus and the Y's were defined to be the proportion of 
successful access attempts using a non-inking stylus. Table 
5 summarized the test and the results from the comparison of 
the inking stylus with the non-inking stylus for the operating 
conditions of lined paper and morning use. Table 5 indicates 
that there is no significant difference between the two 
systems using lined paper in the morning. The significance 
level was 0.01. 

The results from the comparison of the CIC system 
using both an inking and a non-inking stylus on lined paper in 
the afternoon as well as the test itself are summarized in 


Table 6. The results demonstrated that there was not a 
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Significant difference between the two systems using lined 
paper in the afternoon at a significance level of 0.01. 

The next Sign Test performed compared the systems on 
unlined paper in the morning. The test and results are 
summarized in Table 7. These results suggest that there was, 
again, no difference among the systems due to operating 
conditions. The significance level was 0.01. 

The last Sign Test for this set of four compared the 
CIC system with an inking stylus and a non-inking stylus using 
unlined paper in the afternoon. The test and its results are 
again summarized in Table 8. Analyzing the results show that 
there was no significant difference in the systems because of 
the operating conditions. In other words using unlined paper 
in the afternoon did not significantly alter the performance 
of one system over the other. The significance level was 
0.01. 

To summarize the results from this set of Sign Tests, 
there was no significant difference in the performance of the 
CIC system due to the use of an inking or a non-inking stylus 
for the type of paper and the time of use. 

3. The CIC System vs. The Sign/On System for Impostor 

Testing 

The next set of four Sign Tests was used to compare 
the CIC system and the Sign/On system under the different 


operating conditions for the impostor testing portion of the 
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experiment. The definition of a successful access attempt 
changed for the impostor testing. A successful access attempt 
was one in which the system successfully identified an 
impostor and did not grant access. For three of the operating 
conditions none of the impostors were granted access and, 
therefore, the tests and their corresponding results were not 
presented. This was the case for (1) lined paper and morning 
use, (2) lined paper and afternoon use, and (3) Selined paper 
and afternoon use. For the conditions of unlined paper and 
morning use, two impostors were granted access on the Sign/On 
system and one impostor was granted access on the CIC system. 
Table 9 summarized the results and the test for the conditions 
of unlined paper and morning use. There was not a significant 
difference between the two systems under these conditions. 
The significance level was 0.01. 

As before, this set of Sign Tests did not indicate a 
Significant difference in system performance between the CIC 
and the Sign/On systems for impostor testing due to lined or 
unlined paper or due to morning or afternoon use. 

4. The CIC System with Inking Stylus vs. The Sign/On 

System with Non-Inking Stylus For Impostor Testing 

The final set of four Sign Tests comparing the CIC 
system with an inking and a non-inking stylus for impostors 
under the different operating conditions was then performed. 


A total of three impostors were granted access on the CIC 
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system with a non-inking stylus. Two of these impostors were 
granted access in the afternoon on lined paper. The third 
impostor was granted access in the afternoon on unlined paper. 
The impostor who was granted access on the CIC system with an 
inking stylus in the previous phase did not participate in 
this phase of the experiment and, therefore, for this series 
of Sign Tests there were no impostors granted access on the 
CIC system with an inking stylus. Table 10 summarized the 
test and the results for the lined paper and afternoon use of 
the impostor testing. The results indicated that there was no 
Significant difference between the system with an inking 
stylus and a non-inking stylus. The significance level was 
0.01. 

The Sign Test and its results for the impostor testing 
during the afternoon using unlined paper were summarized in 
Table 11. The results indicated that the CIC system using an 
inking stylus did not perform any better than the same system 
with a non-inking stylus. Again, the significance level of 
0.01 was used. 

In summary, this set of four Sign Tests for impostor 
testing did not demonstrate a significant difference in 
performance between the CIC system using an inking or a non- 
inking stylus for the paper conditions and the time 


conditions. 
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5. Sign Tests for Combined Operating Conditions 

By combining all the operating conditions into a 
Single measure for each system, an overall measure for system 
performance will be obtained. In total four Sign Tests were 
completed in this series. The first test compared the CIC 
system with an inking stylus and the Sign/On system. Table 12 
presented the test and the results. Once again the results 
indicated that there was no difference in the performance of 
either system. The significance level was 0.01. 

The next Sign Test for the combined operating 
conditions compared the CIC system using an inking stylus and 
a non-inking stylus. The test and results are summarized in 
Table 13. At a significance level of 0.01, there was no 
difference in the performance of the CIC system due to the 
condition of the stylus. 

Combining the operating conditions for the impostor 
testing was next. Comparing the CIC system and the Sign/On 
system for this Sign Test produced the results in Table 14. 
These results indicated that there was no significant 
difference between the two systems for impostor testing. The 
Significance level was 0.01. 

The last Sign Test for the combined operating 
conditions compared the CIC system using an inking and a non- 
inking stylus for the impostor testing. Table 15 summarized 
the results from that test as well as the test itself. The 


total number of impostors granted access was two. The results 
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indicated that the CIC system with an inking stylus did not 
perform significantly better than the same system with a non- 


inking stylus. The significance level was 0.01. 


C. CONFIDENCE INTERVALS 

In addition to the hypothesis testing done with the Sign 
Tests, confidence intervals were computed to estimate the 
proportion of successful access attempts for the CIC system 
with an inking stylus, the Sign/On system, and the CIC system 
with a non-inking stylus. The confidence intervals were 
computed for both the subjects' access attempts and the 
impostors' access attempts. Because of the large values for 
the proportion of successful access attempts, the Poisson 
approximation to the Binomial was used to compute the three 
confidence intervals for impostor testing (Duncan, 1986, pp. 
100-103). Specifically, the Cumulative Probability Curves for 
the Poisson Distribution were used (Duncan, 1986, pp. 94, 573- 
574). The confidence intervals for the proportion of 
successful access attempts during non-impostor testing, 
however, were computed using the Normal approximation to the 
Binomial because the actual number of authorized subjects 
denied access was greater than 30 (Duncan, 1986, p. 574). The 
six confidence intervals were as follows: 

1. CIC system with inking stylus 


p = proportion of successful access attempts = 0.9862 
CONFIDENCE INTERVAL: Pr(0.981 < p < 0.991) = 0.95% 
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2. Sign/On system 
p = 0.9764 
CONFIDENCE INTERVAL: Pr(0.970 < p < 0.983) 
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3. CIC system with non-inking stylus 
p = 0.9824 
CONFIDENCE INTERVAL: Pr(0.976 < p < 0.989) = 0.95% 
4. CIC system with inking stylus for impostor testing 
p = 0.9995 
CONFIDENCE INTERVAL: Pr(0.998 < p < 1.000) = 0.95% 
5. Sign/On system for impostor testing 
p = 0.9991 
CONFIDENCE INTERVAL: Pr(0.997 < p < 1.000) = 0.95% 
6. CIC system with non-inking stylus for impostor testing 
p = 0.9974 
CONFIDENCE INTERVAL: Pr(0.992 < p < 0.999) = 0.95% 
D. DEPARTMENT OF DEFENSE GUIDELINES 
The DOD has directed the United States Air Force as the 
lead agency to study equipment used to automate physical 
security access control (Bright, 1987, p. 33). Under that 
tasking, the Air Force has provided guidelines for the use of 
automated access control devices. These guidelines define a 
goal of 0.0% for both Type I and Type II errors for an access 
control system. However, the Air Force set the minimum 
requirement for access control systems at 3.0% Type I errors 
and 1.0% Type II errors (Bright, 1987, p. 33). Both the CIC 
system with an inking stylus and a non-inking stylus and the 


Sign/On system exceed those minimums and provide effective 


access control. 
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E. OTHER CONSIDERATIONS 

There are several other factors besides performance which 
impact an access control device. Some of these are the level 
of security required, the cost to operate the system, the 
administrative burden, and the convenience to the user. 

The level of security required is probably the most 
important factor to consider after performance. Highly 
classified material will obviously demand a more stringent 
access control system than personal privacy information. The 
ultimate responsibility for determining the level of access 
control required is set by the individual commanding officer 
(OPNAVINST 5510.1H, 1988, p.2-5). He must be assured by 
performance measures that the access control system is 
adequate for the level of information sensitivity. 

The cost to operate a signature verification system is 
very low. After an initial purchase price, the system's 
operating costs are essentially covered by paper and, in the 
case of the CIC system, ink. The approximate purchase price 
of the CIC system is $1500 while for the Sign/On system the 
purchase price is $1900. These prices, however, are 
approximate due to the fact that each individual system would 
be tailored to fit the security needs of the user. 

The administrative burden associated with access control 
can be considerable. With an access control system relying on 
devices such as passwords or keys the record keeping can be 


burdensome, especially at very large commands. Signature 
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verification systems on the other hand are not an 
administrative burden. Both the CIC system and the Sign/On 
system provide quick and easy administrative maintenance. The 
bulk of administrative tasks would entail enrolling new users 
and deleting users who no longer need access. Each of these 
tasks can be accomplished ina matter of seconds depending on 
the operating capacity of the computer the. signature 
verification system is using. The only other administrative 
concern is the amount of storage needed for authorized users. 
This concern is again a function of the operating capacity of 
the computer the system is using. 

Finally, user convenience is an important factor for 
access control. Signature verification systems are very easy 
to learn and use especially with and effective enrollment 


procedure. 
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IV. CONCLUSIONS AND RECOMMENDATIONS 

The results from this experiment indicated that both the 
CIC On-line Dynamic Signature Verification system using an 
inking stylus and the Sign/On system produced by Capital 
Security Systems Incorporated were effective biometric access 
control devices. The CIC system with an inking stylus hada 
Type I error rate of 1.38% and a Type II error rate of 0.05%. 
The Sign/On system had a Type I error rate of 2.36% and a Type 
II error rate of 0.09%. In other words authorized users were 
denied access 1.38% of the time on the CIC system and 2.36% of 
the time on the Sign/On system and impostors were granted 
access 0.05% of the time for the CIC system and 0.09% of the 
time for the Sign/On system. The small difference in the 
percentages was not statistically significant. Both systems 
exceed the minimum requirements for access control devices set 
by the United States Air Force and as such provide adequate 
security measures for sensitive information. Additionally, 
the experimental results also indicated that the operating 
conditions of lined or unlined paper and morning or afternoon 
use did not significantly affect system performance. These 
results seem to support the claim that the signature is 
similar to a conditioned reflex. 

Department of Defense security requirements have 


necessitated the need for stringent access control. Because 
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of the vulnerabilities of passwords, badges, and keys, 
signature verifications systems. like the CIC system or the 
Sign/On system can provide more reliable and accurate access 
control systems and, thereby, reduce the current threat of 
unauthorized access. Although neither system is perfect, they 
did exceed the published minimum requirements for access 
control and either system is a feasible solution to an 
individual facility's access control problem. The choice 
remains with the facility's commander. 

There are some other areas of concern to be investigated 
prior to widespread implementation of signature verification 
systems used for access control. The first of these is the 
environment in which the system operates. Signature 
verification systems may not be appropriate for deployable 
military units. A second concern is the traffic flow to the 
controlled area of information. The size of the facility may 
have a decided effect on an access control system. A third 
concern is a cost benefit analysis. Finally, verification of 
these experimental results in an operational environment would 
demonstrate the effectiveness of signature verification 


systems for access control. 
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APPENDIX 


SIGN TEST FOR THE CIC SYSTEM vs. 
SYSTEM ON LINED PAPER IN THE AFTERNOON 


SIGN/ON 
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0.800 
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THE SIGN/ON 


DIFFERENCE 


The hypotheses are: 


Ho: The CIC system and the Sign/On system have the same 
proportion of successful access attempts on lined paper 
in the afternoon 


Ha: The CIC system and the Sign/On system do not have 
the same proportion of successful access attempts on 


lined paper in the afternoon 
0.01 


The observed number of plus signs is 3 out of a possible 
6. From any Binomial Distribution Table, the 
corresponding probability level is 0.6562. 


Because the probability level is greater than a, one 
CANNOT REJECT THAT THE PROPORTION OF SUCCESSFUL ACCESS 
ATTEMPTS FOR BOTH SYSTEMS ARE THE SAME ON LINED PAPER IN 
THE AFTERNOON AT A SIGNIFICANCE LEVEL OF O.O1. 


* only subjects with differences not equal to O were 
considered 
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Table 3. SIGN TEST FOR THE CIC SYSTEM vs. THE SIGN/ON 
SYSTEM ON UNLINED PAPER IN THE MORNING 


CIC SIGN/ON DIFFERENCE 
1.000 0.958 
1.000 0.833 
1.000 G97 
1.000 0.958 
0.958 0.917 
0.917 1.000 
0.917 0.875 
1.000 0.917 
0.958 1.000 
1.000 0.958 
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The hypotheses are: 


Ho: The CIC system and the Sign/On system have the same 
proportion of successful access attempts on unlined 
paper in the morning 


Ha: The CIC system and the Sign/On system do not have 
the same proportion of successful access attempts on 
unlined paper in the morning 


0.01 


The observed number of minus Signs iS 2 out of a 
possible 10. From any Binomial Distribution Table, the 
corresponding probability level is 0.0547. 


Because the probability level is greater than a, one 
CANNOT REJECT THAT THE PROPORTION OF SUCCESSFUL ACCESS 
ATTEMPTS FOR BOTH SYSTEMS ARE NOT THE SAME ON UNLINED 
PAPER IN THE MORNING AT A SIGNIFICANCE LEVEL OF 0O.O1. 


* only subjects with differences not equal to O were 
considered 
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Table 4. 


oe 


rPOUOAONA MN PWNE 


cic 
1.000 
0.958 
1.000 
0.950 
0.958 
1.000 
0.958 
1.000 
0.875 
0.958 
0.958 


SIGN/ON 


0.958 
1.000 
0.875 
1.000 
1.000 
O02. 952 
0.875 
0.958 
1.000 
1.000 
1.000 


SIGN TEST FOR THE CIC SYSTEM vs. THE SIGN/ON 
SYSTEM ON UNLINED PAPER IN THE AFTERNOON 


DIFFERENCE 


a pate ee 6 


r++ 


The hypotheses are: 


Ho: The CIC system and the Sign/On system have the same 
proportion of successful access attempts on unlined 
paper in the afternoon 


Ha: The CIC system and the Sign/On system do not have 
the same proportion of successful access attempts on 
unlined paper in the afternoon 


0.01 


The observed number of plus signs is 5 out of a possible 
diac From any Binomial Distribution Table, the 
corresponding probability level is 0.5000. 


Because the probability level is greater than a, one 
CANNOT REJECT THAT THE PROPORTION OF SUCCESSFUL ACCESS 
ATTEMPTS FOR BOTH SYSTEMS ARE THE SAME ON UNLINED PAPER 
IN THE AFTERNOON AT A SIGNIFICANCE LEVEL OF 0.01. 


* only subjects with differences not equal to O were 
considered 
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Table 5. SIGN TEST FOR THE CIC SYSTEM WITH INKING STYLUS 
vs. NON-INKING STYLUS ON LINED PAPER IN THE MORNING 


SUBJECT” INKING NON-INKING DIFFERENCE 
1 1.000 0.958 +; 
2 Oe d7 Oo 50 aE 
3 O.2752 1.000 = 
4 O78 7/3 1.000 = 


The hypotheses are: 


Ho: The CIC system using an inking stylus and a non- 
inking stylus have the same proportion of successful 
access attempts on lined paper in the morning 


Ha: The CIC system using an inking stylus and a non- 
inking stylus do not have the same proportion of 
successful access attempts on lined paper in the morning 


0.01 


The observed number of plus signs is 2 out of a possible 
AW From any Binomial Distribution Table, the 
corresponding probability level is 0.6875. 


Because the probability level is greater than a, one 
CANNOT REJECT THAT THE PROPORTION OF SUCCESSFUL ACCESS 
ATTEMPTS FOR THE SYSTEM IS THE SAME WHETHER OR NOT AN 
INKING STYLUS IS USED ON LINED PAPER IN THE MORNING AT 
A SIGNIFICANCE LEVEL OF 0.01. 


* only subjects with differences not equal to O were 
considered 
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Table 6. SIGN TEST FOR THE CIC SYSTEM WITH INKING STYLUS 
vs. NON-INKING STYLUS ON LINED PAPER IN THE AFTERNOON 


INKING NON-INKING DIFFERENCE 
0.950 1.000 
0.958 0.833 
OF gi 7 1.000 


The hypotheses are: 


Ho: The CIC system using an inking stylus and a non- 
inking stylus have the same proportion of successful 
access attempts on lined paper in the afternoon 


Ha: The CIC system using an inking stylus and a non- 
inking stylus do not have the same proportion of 
successful access attempts on lined paper in the 
afternoon 


0.01 


The observed number of plus signs is 1 out of a possible 
Si From any Binomial Distribution Table, the 
corresponding probability level is 0.5000. 


Because the probability level is greater than a, one 
CANNOT REJECT THAT THE PROPORTION OF SUCCESSFUL ACCESS 
ATTEMPTS FOR THE SYSTEM IS THE SAME WHETHER OR NOT AN 
INKING STYLUS IS USED ON LINED PAPER IN THE AFTERNOON AT 
A SIGNIFICANCE LEVEL OF 0.01. 


* only subjects with differences not equal to O were 
considered 
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Table 7. SIGN TEST FOR THE CIC SYSTEM WITH INKING STYLUS 
vs. NON-INKING STYLUS ON UNLINED PAPER IN THE MORNING 





SUBJECT” INKING NON-INKING DIFFERENCE 
1-000 0.850 
1.000 0.950 
0.900 On792 
0.958 1.000 
0.958 1.000 


The hypotheses are: 


Ho: The CIC system using an inking stylus and a non- 
inking stylus have the same proportion of successful 
access attempts on unlined paper in the morning 


Ha: The CIC system using an inking stylus and a non- 
inking stylus do not have the same proportion of 
successful access attempts on unlined paper in the 
morning 


0202 


The observed number of minus signs is 2 out of a 
possible 5. From any Binomial Distribution Table, the 
corresponding probability level is 0.5000. 


Because the probability level is greater than a, one 
CANNOT REJECT THAT THE PROPORTION OF SUCCESSFUL ACCESS 
ATTEMPTS FOR THE SYSTEM IS THE SAME WHETHER OR NOT AN 
INKING STYLUS IS USED ON UNLINED PAPER IN THE MORNING AT 
A SIGNIFICANCE LEVEL OF 0.01. 


* only subjects with differences not equal to O were 
considered 


a7 


Table 8. SIGN TEST FOR THE CIC SYSTEM WITH INKING STYLUS 
vs. NON-INKING STYLUS ON UNLINED PAPER IN THE AFTERNOON 


SUBJECT’ INKING NON-INKING DIFFERENCE 
ih 0.958 1.000 
2 0.958 0.625 
g 0.958 1.000 
4 0.958 1.000 


The hypotheses are: 


Ho: The CIC system using an inking stylus and a non- 
inking stylus have the same proportion of successful 
access attempts on unlined paper in the afternoon 


Ha: The CIC system using an inking stylus and a non- 
inking stylus do not have the same proportion of 
successful access attempts on unlined paper in the 
afternoon 


0.01 


The observed number of plus signs is 1 out of a possible 
4. From any Binomial Distribution Table, the 
corresponding probability level is 0.3125. 


Because the probability level is greater than a, one 
CANNOT REJECT THAT THE PROPORTION OF SUCCESSFUL ACCESS 
ATTEMPTS FOR THE SYSTEM IS THE SAME WHETHER OR NOT AN 
INKING STYLUS IS USED ON UNLINED PAPER IN THE AFTERNOON 
AT A SIGNIFICANCE LEVEL OF 0.01. 


* only subjects with differences not equal to 0O were 
considered 
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Table 9. SIGN TEST FOR IMPOSTOR TESTING FOR THE CIC SYSTEM 
vs. THE SIGN/ON SYSTEM ON UNLINED PAPER IN THE MORNING 


Cie SIGN/ON DIFFERENCE 
1.000 0.958 + 
1.000 0.958 AF 
0.958 1.000 = 


The hypotheses are: 
Ho: The CIC system and the Sign/On system have the same 


proportion of successful access attempts for impostor 
testing on unlined paper in the morning 


Ha: The CIC system and the Sign/On system do not have 
the same proportion of successful access attempts for 
impostor testing on unlined paper in the morning 


Oi OE 


The observed number of minus signs is 1 out of a 
possible 3. From any Binomial Distribution Table, the 
corresponding probability level is 0.5000. 


Because the probability level is greater than a, one 
CANNOT REJECT THAT THE PROPORTION OF SUCCESSFUL ACCESS 
ATTEMPTS FOR BOTH SYSTEMS FOR IMPOSTOR TESTING ARE THE 
SAME ON UNLINED PAPER IN THE MORNING AT A SIGNIFICANCE 
LEVEL OF 0.01. 


* only subjects will differences not equal to O were 
considered 
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Table 10. SIGN TEST FOR THE CIC SYSTEM WITH AN INKING 
STYLUS vs. NON-INKING STYLUS FOR IMPOSTOR TESTING ON LINED 
PAPER IN THE AFTERNOON 


INKING NON-INKING DIFFERENCE 
1.000 0.944 ae 
1.000 0.944 =F 


The hypotheses are: 


Ho: The CIC system using an inking stylus and a non- 
inking stylus for impostor testing have the same 
proportion of successful access attempts on lined paper 
in the afternoon 


Ha: The CIC system using an inking stylus and a non- 
inking stylus for impostor testing do not have the same 
proportion of successful access attempts on lined paper 
in the afternoon 


0.01 


The observed number of minus signs is 0O out of a 
possible 2. From any Binomial Distribution Table, the 
corresponding probability level is 0.2500. 


Because the probability level is greater than a, one 

CANNOT REJECT THAT THE PROPORTION OF SUCCESSFUL ACCESS 
ATTEMPTS FOR THE SYSTEM IS THE SAME WHETHER OR NOT AN 
INKING STYLUS IS USED FOR IMPOSTOR TESTING ON LINED 
PAPER IN THE AFTERNOON AT A SIGNIFICANCE LEVEL OF 0O.O1. 


* only subjects with differences not equal to O were 
considered 
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Table 11. SIGN TEST FOR THE CIC SYSTEM WITH INKING STYLUS 
vs. NON-INKING STYLUS FOR IMPOSTOR TESTING ON UNLINED PAPER 
IN THE AFTERNOON 


SUBJECT" INKING NON-INKING DIFFERENCE 
1 1.000 0.944 a9 


The hypotheses are: 


Ho: The CIC system using an inking stylus and a non- 
inking stylus have the same proportion of successful 
access attempts for impostor testing on unlined paper in 
the afternoon 


Ha: The CIC system uSing an inking stylus and a non- 
inking stylus have the same proportion of successful 
access attempts for impostor testing on unlined paper in 
the afternoon 


0.01 


The observed number of minus signs is 0O out of a 
possible 1. From any Binomial Distribution Table, the 
corresponding probability level is 0.5000. 


Because the probability level is greater than a, one 

CANNOT REJECT THAT THE PROPORTION OF SUCCESSFUL ACCESS 
ATTEMPTS FOR THE SYSTEM IS THE SAME WHETHER OR NOT AN 
INKING STYLUS IS USED FOR IMPOSTOR TESTING ON UNLINED 
PAPER IN THE AFTERNOON AT A SIGNIFICANCE LEVEL OF 0.01. 


* only subjects with differences not equal to O were 
considered 
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Table 12. 
SYSTEM 


SUBJECT’ 


OMAN ANF WN FP 


The hypotheses are: 


CIC 
1.000 
1.000 
0.934 
1.000 
0.955 
1.000 
0.989 
0.969 
1.000 
0.938 
Ooo 
0.969 
0.969 
0.948 
1.000 


SIGN/ON 


0.966 
0. 230 
0.948 
02269 
0.920 
0.960 
0. 920 
1.000 
0.990 
0.927 
0.938 
02990 
1.000 
0.969 
0.938 


SIGN TEST FOR THE CIC SYSTEM vs. THE SIGN/ON 


DIFFERENCE 


Pe+etettteeutt 


+ 


Ho: The CIC system and the Sign/On system have the same 
proportion of successful access attempts 


Ha: The CIC system and the Sign/On system do not have 
the same proportion of successful access attempts 


a = 0.01 


The observed number of minus signs is 5 out of a 
From any Binomial Distribution Table, the 
corresponding probability level is 0.1509. 


possible 15. 


Because the probability level is greater than a, one 

CANNOT REJECT THAT THE PROPORTION OF SUCCESSFUL ACCESS 
ATTEMPTS FOR BOTH SYSTEMS ARE THE SAME AT A SIGNIFICANCE 
LEVEL OF QO.01. 


* only subjects with differences not equal to O were 
considered 
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Table 13. SIGN TEST FOR THE CIC SYSTEM WITH INKING STYLUS 
vs. NON-INKING STYLUS 


SUBJECT’ INKING NON-INKING DIFFERENCE 


SOO WN EF 


0.958 
0.989 
1.000 
0.938 
02979 
0.969 
0.948 


Oe 927 
1.000 
0.989 
O77 
P0000 
1.000 
1.000 


ale 


The hypotheses are: 


Ho: The CIC system using an inking stylus and a non- 
inking stylus have the same proportion of successful 
access attempts 


Ha: The CIC system using an inking stylus and a non- 
inking stylus do not have the same proportion of 
successful access attempts 


0.01 


The observed number of plus signs is 3 out of a possible 
ae From any Binomial Distribution Table, the 
corresponding probability level is 0.5000. 


Because the probability level is greater than a, one 
CANNOT REJECT THAT THE PROPORTION OF SUCCESSFUL ACCESS 
ATTEMPTS FOR THE SYSTEM IS THE SAME WHETHER OR NOT AN 
INKING STYLUS IS USED AT A SIGNIFICANCE LEVEL OF 0.01. 


* only subjects with differences not equal to O were 
considered 





43 


Table 14. SIGN TEST FOR THE CIC SYSTEM vs. THE SIGN/ON 
SYSTEM FOR IMPOSTOR TESTING 


CIC SIGN/ON DIFFERENCE 
1.000 Oe 989 Te 
1.000 0.989 + 
0.989 1.000 = 


The hypotheses are: 


Ho: The CIC system and the Sign/On system have the same 
proportion of successful access attempts for impostor 
testing 


Ha: The CIC system and the Sign/On system do not have 
the same proportion of successful access attempts for 
impostor testing 


0.01 


The observed number of minus signs is 1 out of a 
possible 3. From any Binomial Distribution Table, the 
corresponding probability level is 0.5000. 


Because the probability level is greater than a, one 
CANNOT REJECT THAT THE PROPORTION OF SUCCESSFUL ACCESS 
ATTEMPTS FOR BOTH SYSTEMS ARE THE SAME FOR IMPOSTOR 
TESTING AT A SIGNIFICANCE LEVEL OF 0.01. 


* only subjects with differences not equal to O were 
considered 
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Table 15. SIGN TEST FOR THE CIC SYSTEM WITH INKING STYLUS 
vs. NON-INKING STYLUS FOR IMPOSTOR TESTING 


INKING NON-INKING DIFFERENCE 
1.000 0.985 + 
1.000 O29 71 + 


~, 


The hypotheses are: 


Ho: The CIC system using an inking stylus and a non- 
inking stylus have the same proportion of successful 
access attempts for impostor testing 


Ha: The CIC system using an inking stylus and a non- 
inking stylus do not have the same proportion of 
successful access attempts for impostor testing 


O01 


The observed number of minus Signs 1S 0 out of a 
possible 2. From any Binomial Distribution Table, the 
corresponding probability level is 0.2500. 


Because the probability level is greater than a, one 
CANNOT REJECT THAT THE PROPORTION OF SUCCESSFUL ACCESS 
ATTEMPTS FOR THE SYSTEM IS THE SAME WHETHER OR NOT AN 
INKING STYLUS IS USED FOR IMPOSTOR TESTING AT A 
SIGNIFICANCE LEVEL OF 0.01. 


* only subjects with difference not equal to O were 
considered 
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